posted 12-30-2011 08:26 PM         

A private examiner gets an inquiry that goes something like this...

"I'm applying for a position with the FBI [or Podunk, GA PD/Raytheon/Noblis, etc.] and I've heard about 50% of applicants get eliminated via the polygraph process. So, before I get too heavily invested, I want to see how I might do on the polygraph."

o Would taking such a case be a violation of APA policy?

o If taking the case is not an APA violation, is it an ethical conflict? If so, why?

posted 01-01-2012 09:43 PM         

Still, this could be a solid marketing opportunity for private examiners. I suspect that many a SA/LE/contractor applicant is spooked by what he/she has heard about the polygraph to the extent that he/she will pay -- perhaps handsomely -- to have the process fully explained and totally demystified over a battery of polygraph sessions. Hence, my question centers on the APA's official position on the ethics of such private testing (similar to fidelity exams), not on the "practice" test's efficacy per se.

Dan

posted 01-03-2012 06:34 AM            

This is the right time to have this discussion, though, before anyone actually goes out to market the idea. We need to go into this with eyes open if we are to do it at all. I appreciate you starting the conversation.

posted 01-03-2012 08:10 AM            

If we need a policy on this type of thing, that policy should be based on what is best for the community and what is best for the profession. The assumption should be that these are not mutually exclusive. What is best for the community is best for the profession. Membership is a professional association is a form of agreement that adherence to what is best for the community and the profession will also ultimately be best for individual professionals.

Good policies should ensure the future credibility of the profession and the future livelihood of good professionals by ensuring the community of the highest standards of work and professional conduct (in practice this is done by setting minimum standards, and minimum standards too often become the maximum that some strive to achieve).

So, a guiding question would be the one Don raises. Would this practice provide the best benefit to the community, to the profession, and to individual professionals?

Don raises a second issue: what is the difference between a practice test and coaching? The phrase "fully explained and totally demystified over a battery of polygraph sessions" sounds a lot like coaching, and does not seem to fit the form of a properly conducted private exam.

Don raises a third issue more indirectly: does it change the accuracy of the polygraph test to engage in a practice or preparatory exam? Said differently: are there differences in the normative data when an examinee is coached? Answer: maybe, we don't really know, but quite possibly.

Here is another question: what standardized scientific test require coaching. Answer: none. (Keep in mind that in the realm of science "standardized" means "normed.") Normed (standardized) scientific tests are designed and validated to provide accuracy when the response to the test stimuli are authentic. Certain forms of preparation are OK - such as an LSAT prep course. Howevever, coaching, in the form of teaching (demystifying) the test questions, test designe, and test structure can only be expected to deplete the accuracy of the test. Sure it may increase test scores for some people - but the accurate meaning of those test scores will be lost.

Same with the MMPI - another standardized (normed) test for which excessive preparation is not necessary - beyond basic rest, nutrition, and orientation to the need and purpose for the test. Too much information about the test itself can be expected to confound our ability to interpret the meaning of the test data.

Standardized tests of credibilty may be not much different than other standardized tests in terms of the scientific issues that need to be navigated and managed.

For the sake of argument (and not intended to represent the views of anyone or any entity other than myself), I will argue the practice of coaching to be contrary to the interests of the community, the profession, and professionals interested in a a long and viable future.

Why pee in our own pool?

As always,

my .02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-03-2012 02:23 PM            

As always, your point is well taken, but how is the private examiner to know the real reason for the test? And if they do know the examinee is using the test as practice, should there be a prohibition against the test. If I’m working as a private examiner, which I am not, and a client comes to me and wants to be examined because they are considering applying for a particular job, should I be permitted to run the test? Just because my client does well on my test doesn’t mean they will do well on the test administered by the employing agency. As a private examiner, I think I am making an assumption that I know how the employing agency is going to conduct their test (format, test data analysis, questions used). As with a defense attorney, just because they pass the defense test doesn’t mean that they will pass the LE test. In theory they should, but it isn’t a guarantee. I have had it go both ways.

The same could happen in Dan’s hypothetical. I think private examiners are often working with a lot of unknowns, but they work with the information that they have. Private examiners are often working at a disadvantage, but they are trying to provide a service to their client.

Dan’s ethical situation might be, What did the examiner know, and when did he/she know it?

posted 01-03-2012 05:10 PM            

One way to gain information about the real reason for a test is to question the referral about the referral question.

For example, every psych-eval has a referral question. The question can be something basic such as is this person suitable for law enforcement? Or, a question can be more specific such as what are patient's diagnosis and treatment needs? Or, a referral can be very specific, such as whether the person's neuropsychological functioning, including cognitive and motor skills are suitable for piloting a jet aircraft. A question can be post-incident, such as what is the nature of the person's neuroligical deficits subsequent to some form of trauma.

The point is that a credible referral will come with credible questions. Then, the information and results will go to the referral source.

If all of the information comes from an attorney who refuses to provide any form of written detail - even if limited to the attornye's own synopsis of the known details of an open investigation regarding a known incident or known allegation (for which the police reports are commonly not available) - then the referral is suspect, including the referral information and the referral question. There is no great problem, in my view, to conduct an exam with written details from an attorney - when the attorney states in writing the facts as they are known. It will become evident later if the attorney has misrepresented the case at the time of the referral - and that could be a potential concern to the Bar Association.

Even more hazardous is the situation of self-referrals. If all of the referral information comes from the examinee, and all of the test information and results go to the examinee - then there are perhaps big problems.

A credible referral comes from a credible referral source with credible referral questions. This may be the difference between legitimate attorney referrals and some form of sham.

Just sayin'

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-03-2012 08:45 PM         

"There you go again". It is though you are angry at the APA and the polygraph profession. What gives here? If you don't like the work that you do, or the advice and counsel from the people in your profession- find a new profession!

We are all trying to get polygraph on the scientific map and get the rest of the world to accept our profession. Are you in- or are you out? Again, you are doing nothing more than pissin' and moanin'. How about some real solutions or ideas from you?

Ted

[This message has been edited by Ted Todd (edited 01-03-2012).]

posted 01-03-2012 09:18 PM            

But, from Ray’s explanation, as I understand his explanation, Dan’s hypothetical would be a self referral and there could be problems (set me straight if I’m wrong, Ray).

posted 01-03-2012 10:02 PM            

Self-referred exams bother me. A self-referred psych-eval would bother me too. Why? Who benefits and how? Of what use is a complicated test result like a psych-eval or polygraph result (not that they are the same - but they are both complicated because it is always more than just a bunch of numbers - their are always pieces of information that are rich and useful beyond numbers) - of what use is this to the person. Normally, results and information of this type serves a decision support role for someone having to make difficult decisions. So, it gives the impression of some kind of hijacked process.

Imagine a police applicant who hires his own background investigator. I think most departments and agencies would prefer to do their own investigations or contract with the investigator themselves.

Consider this: what is the possibility that these kinds of subversive acts, like countermeasures, may not be all that effective - that, like countermeasures, the polygraph might work at rates that are significantly greater than chance even in the context of these problems. If so, then our real responsibility is simply to be aware of, but not crazy about the potential problem, and to clarify the ethical and professional boundaries around coaching so that we can both reduce and highlight its occurrence.

One key illustrator, in my opinion, is when a polygraph report reads like this: "On X data, at your request, I conducted a polygraph exam on you..."

There must be some way to safely structure these kinds of practices.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-04-2012 07:58 AM         

What you're admitting here is that the polygraph's accuracy depends on secrecy. And polygraph testing is "scientific"...how?

Some other questions:

What is the rate of FP results for federal and LE applicants?

Why don't all federal agencies record applicant (and other) polygraphs? Are they exempt from APA standards?

Are federal/LE applicant tests, once conducted, available for independent QA review? Why not?

Does a cautious, perhaps slightly paranoid federal applicant -- after being spooked by what he's read on AP.org -- have a right to a private polygraph to simply "see how he'll do" before he gets too invested in the process?

posted 01-04-2012 08:58 AM            

What is the rate of FP results for federal and LE applicants?
A: No reason to assume they are any higher or lower than the FP rates of other tests conducted under similar circumstances. Hiring decisions in the federal government or law enforcement involve much more than the results of a polygraph. Disqualifying admissions during the polygraph test result in most rejections, not the polygraph results alone.

Why don't all federal agencies record applicant (and other) polygraphs? Are they exempt from APA standards?

A: APA does not require recording of all polygraph examinations in the standards. As an example, all Army CID criminal examinations are monitored by an agent but not recorded but there is a government witness watching the entire examination.

Are federal/LE applicant tests, once conducted, available for independent QA review? Why not?

A: All federal agencies have a QC process that does QC reviews of their examinations. Do you really think the CIA/NSA/FBI is going to send their exams out to a private examiner for review?

posted 01-04-2012 07:31 PM         

But, I know, that’s just the way it is. Collateral damage. Too bad, so sad. No one owes them a job. After all, the polygraph isn’t a scalpel, it’s a sledgehammer. (But it’s scientific, right?) “Next please!”

Polygraph is an industry. That industry has dynamics of supply and demand. Part of the demand is created by examiner incompetence in the form of false positive results. Those who are adversely affected by FP results should have a right to a second opinion. Those who legitimately fear a FP result should have the right to a dry run.

Your fear of the sinister bogeyman comes at a cost that’s borne by legitimate victims. Those individuals who are aggrieved in such a manner deserve a remedy. That’s the American way. Evidently you are determined to put a choke hold on an industry that could otherwise enjoy robust growth by exploiting market potential.

If practice testing takes off like I think it will, then the examiners on your side of the fence will simply have to up their game. Those of us who do PCSOT tests on very well informed frequent flyers have been doing that for years. Why can’t you?

posted 01-04-2012 08:13 PM            

I know you asked Skip, but I'll chime in re the FP rate of screening exams.

We will never know the actual FP rate because it will be impossible to confirm every case. Its the same as knowing the actually accuracy of the polygraph - it would require conducting every possible test on every person and confirming every result.

What we can actually achieve is scientific (statistical) calculations of an estimate of test accuracy along with a mathematical calculation of the error range of the statistic.

Nothing is perfect. What we attempt to do is quantify the range of imperfection.

The recent meta-analysis indicates that published and replicated studies on PDD techniques for which the results are interpreted with decision rules based on an assumption that the criterion variance of the RQs is independent (read: multi-issue) show that the FP rate is about about 14.4%, with at standard error of 3.9%. This gives a 95% confidence interval of 6.6% to 22.1%.

Those techniques are the TES/DLST and the AFMGQT.

Is this number perfect? Of course not. But it is a reasonable evidence-based estimate for now. And it is not simply a number that someone made up based on their anecdotal experience, a personal belief system, charismatic personality, or forceful opinion. It's data. Sure it's actually blunt and messy. But we attempt to be precise so that we refrain from simply making up fiction. And while we attempt to be precise about our estimate, we also do our best to calculate the likely range of error within which the result will occur 95% of the time if we could re-do the analysis over and over and over with different data.

We can quibble all day long about the semantics of the test questions. The bottom line is that semantic arguments are usually about as testable as Freudian psychoanalysis - which means these arguments are rather un-scientific. What we should actually assume (and require) is that test questions be formulated with sufficient logical and linguistic clarity (simplicity) that the examinee knows WTF the question is asking and knows whether or not he or she is lying about his or her past behavior as described by the question. That is all. If they know what the question asks, and what the truth is then it should work. Period.

Is 22% FP kinda high? Maybe. Maybe not.

Both test sensitivity and specificity are significantly greater than chance for some of our screening techniques. This means that deceptive people are more likely to get excluded while using the polygraph. DUH!

It also means that truthful people are MORE likely to be hired while using the polygraph, than if the polygraph were not used.

Same with sex offenders. People who cannot follow rules are more likely to be remanded to more structured instituional settings, and those who can follow rules are more likely to be allowed to remain in the community.

We could calculate all this too, based on the sensitivity and error rates, but that's another posting.

The point is that the scientific evidence does seem to support the suggestion that polygraph screening - even multi-issue screening does actually help truthful people. If we do it right - using techniques that are proven to work. If we use unproven methods then it is of course more difficult to know - and a lot more difficult to answer the questions and challenges of our critics. It will also be more difficult to win the hearts and minds of legislators and policy makers that have to decide whether or not they want to spend a part of their budgets on polygraph services.

If we stick to the science, answer good questions with evidence, use methods that work, depricate those that do not, and refrain from the simplistic hyperbole that interferes with our ability to understand and describe the validity of our work - then it will all be easier in the future, and we will enjoy the opportunity to provide a highly valued service to our agencies, communities, and countries.

Peace,

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-04-2012 10:11 PM            

First there are two differing issues here. One, does a defense attorney get a private exam for his client accused of a crime; yes! This happens all the time. The first reason is to see if he will pass and as Robert indicated the examiner usually only has what the attorney wants to give them; thereby setting the client up to fail or improperly pass. The other reason is to get their client to accept a deal or negotiate a plea.

Regarding applicants, this is already a common occurence. Many applicants apply at multiple agencies and coordinate their testing to process from their least desired to most desired position, thereby "practicing" their techniques in polygraph, interviewing, MMPI, etc.

When they arrive for their truly desired position, they come off calmer, more confident and very well practiced. There is no reason to pay a private examiner for a practice test, they get all the practice they want for free with municipal and state agencies.

There is no way to stop this practice and trying to regulate this would be impossible.

On a side note, there are still good applicants out there who don't lower themselves to such practices.

Just my thoughts,

Jim

posted 01-05-2012 06:44 AM            

It has been a couple of decades since I left private practice, but I think I still understand the need to find new markets to keep the lights on and the family fed. May I suggest to you that a better alternative is to investigate what the APA is calling "community safety" testing. This is where the polygraph is used to monitor individuals who are convicted of some offense to ensure compliance with probation and parole. The most well known among these types of exams is the PCSOT, but there are other avenues. Rob Lundell is doing a lot of work for the courts in the area of monitoring those convicted of domestic violence. John Swartz (the non-fed one) in Texas has made serious inroads in testing those on probation for drunk driving. These guys are not just making a living, but their work directly prevents injury and deaths of innocent people. They also get virtually 100% confessions in DI tests. And, the controversy over this type of work is really quite low since the public can see that it serves the greater good. I can send you some material, if it interests you.

posted 01-05-2012 08:44 AM            

How will practice with you help them? Are you planning to test in an incompetent manner to replicate those inept fed/le examiners? If you are going to run a competent test on them, then your results would be of little value to them if the next guy is so incompetent that he will run a poor test. They would be given false hope by your highly competent testing only to have those hopes dashed by the imbecile who next tests them for real.

I really don't have a dog in this hunt as I only conduct criminal specific tests on suspects using the ZCT or the modified AFMGQT all of which are subjected to "blind" quality control review. I'm sure I still run the occasional FP as I'm also sure you do as well. If you profess that you don't then the probability is that you are running the occasional FN test.

posted 01-06-2012 10:05 AM         

Wake up and smell the decade. Doug Williams perfected and captured the "Practice Test" market years ago. I have several copies of his CDs if you are interested on how he did it!

Ted

posted 01-06-2012 06:01 PM         

Just kidding.

I've long advocated for the victims of false positive results, hence my early and tireless support of the Quadri-Track technique. (Even if it does rely on "complex" rules, "fancy" questions and is practiced in near secrecy by only a "cloistered" few. Right, Ray?) Had a few epic battles about it on this very board some years back, as I recall. Good times.

BTW, did you guys hear the Quadri-Track was replicated and validated? Amazing.

posted 01-07-2012 12:03 AM            

quote:

---

BTW, did you guys hear the Quadri-Track was replicated and validated? Amazing.

---

posted 01-07-2012 05:20 PM            

Thanks for the very interesting discussion topic.

I assume that all this was just a fun provocative way of poking a stick in our eye.

Regardless of differences, the discussion is interesting and informative.This place has been morgue-like for a while.

So.. thanks, and Happy New Year to you.

BTW, the MQTZCT is included in the meta-analysis on validated techniques because it is supported by multiple published (replicated) studies describing the validity and reliability of the technique. (How's that for a party-line?) Anyone who is properly trained and confident in using this technique has information to cite and refer to in the event of some eventual need to answer challenges or questions regarding the published validation data.

There is a lot to learn from reviewing the published literature.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-07-2012 07:24 PM         

A stick-in-the-eye exercise just for fun? Hardly. The practice test market has great potential.

Ironically, I got the idea from a retired federal examiner (no BS) who does these tests. He laid it all out for me: The conventional wisdom among the general public is that the polygraph is about as accurate as flipping a coin, based on the individual. Therefore, it's simply a matter of identifying a qualified market (in this case, motivated Fed/LE/contractor test-takers), exploit their fear of the 50/50 factor (by citing studies, horror stories, etc.), and then make the sale that a practice/familiarization test is in their best interest to see how "they" (as individuals) will fare in the coin flip.

There is no "coaching," just a thumbs up or thumbs down on the test results. No charts are shown or released. The demystification, such as it is, occurs by osmosis -- mere repetition of the process in a totally neutral and professional setting, without any pressure whatsoever.

It's lucrative, too. Sure, a candidate with concern about one question might be handled in a single polygraph session. But a candidate who's worried about anything and everything might require several days of testing. Indeed they PAY, so I'm told.

Clearly, this is totally opposite from the nickel-and-dime 99-question Jerry Springer type of prospective clients that "retail" polygraph usually attracts. Marketed correctly, practice tests could really give a much-needed boost to the polygraph industry.

And Ted, I did wake up and smell the decade. It's still the 1930s. Don't believe me? I suggest that you read John Larson's 400+ page work "Lying and Its Detection," originally published in 1932. Aside from Reid's control question and Backster's standardization of scoring, not that much has changed since then.

The reality is this: Even the most ardent attempts at 21st-century statistical alchemy will never shore up polygraph's feeble foundation enough to pass scientific muster in the forensic arena.

posted 01-07-2012 08:14 PM         

Best,

Ted

[This message has been edited by Ted Todd (edited 01-07-2012).]

posted 01-08-2012 11:56 AM         

In your original reply you wrote something like "...and then you could send your graduates to me for a REAL polygraph test!"

Do I have the gist of it, Ted?

In any event, my good man, I believe you have put forth a solid polygraph marketing idea in your own right: "The Practice Test Validation Test"

Ted, it's positively brilliant. We could ping-pong these clients back and forth until their credit cards are maxed out. Hell, we could be co-presenters at APA seminars,co-author articles, and do lots more to cash in. Ted, we could OWN this action! Maybe franchise it! (Ya gotta think BIG!)

But I am curious... What would your "real" polygraph test include that mine doesn't? A healthy dose of swaggering bravado in the pre-test phase? Grim theatrics to help "set" the controls? Over-the-top histrionics to coerce admissions? Maybe I'm missing something...something remotely scientific. Please enlighten us.

By the way Ted, mine is bigger than yours. (Polygraph library, that is.) As for your "take a training course" dig, I'm happy to tell you that I am in continual compliance with the 30-hour (bi-annually) rule, with the emphasis on PCSOT. (Actually, there was a lag in '10-'11 due to some course seminar cancellations at Backster, but I caught up ASAP.) I'd love to attend more courses and seminars, but it's a financial hardship for me to do so. There's not much action in New England, and tuitions, air fare, hotels, meals, etc. really add up. To make matters worse, my skinflint yankee agency has zero training dollars for out-of-state events.

Ted, in all seriousness, remember that polygraph is an industry. Competition is good for industry. Examiners compete, polygraph schools compete, instrument makers compete, insurance providers compete, and so on. It's natural and healthy. If the practice test market were to flourish, it would -- by necessity -- eventually help foster a higher level of proficiency straight across the board.

Dan

posted 01-08-2012 12:37 PM         

Take care,

posted 01-08-2012 05:19 PM            

quote:

---

The reality is this: Even the most ardent attempts at 21st-century statistical alchemy will never shore up polygraph's feeble foundation enough to pass scientific muster in the forensic arena.

---

What does that mean? Polygraph is used more in the "forensic arena" (i.e. in law enforcement, for forensic and non-forensic purposes) than anywhere else.

Didn't the NAS refer to it as a forensic science? Why fight against it if the scientific community has recognized it? Sure, they didn't say it was perfect (but what test is?), but even at their figures, we're ahead of eyewitness evidence.

quote:

---

What would your "real" polygraph test include that mine doesn't?

---

That was my first point: you can't provide the service your imaginary client requested. You can only run a test without even knowing what the questions will be. What issues did the background investigator question for which there was no other means of verification? To whom did the candidate lie in the process if there isn't a complete process yet? My initial take is that you'd need to offer a service you couldn't be sure you could provide.

posted 01-09-2012 11:09 AM            

quote:

---

You seem to be arguing that accuracy estimates differ for reasons other than random sampling error. If that's the case (and I'd agree), then what does a practice test that could be vastly different from Ted's real test, offer a person?

---

I do not agree that accuracy differs for reasons other than random error.

The recent meta-analysis shows this and states this. Differences in the accuracy rates of all validated polygraph techniques are not statistically significant. This means that we are NOT sure (evidence does not support) that accuracy differs for any reason other than random sampling error (uncontrolled variance).

Dan: I happen to like statistical "alchemy." Without statistical evidence, we have nothing to base or conclusions on except opinion or individual case anecdotes. Case anecdotes have been historically misleading in many fields - so we have to remain concerned with what happens for MOST (not individual) cases. Opinion too easily degrades to a test of either charm and personality, ego, or power and politics (which is too often about econcomic$).

If we, as professionals, want to be all about the truth, then there is nothing more correct than to be interested in evidence-based scientific approaches to the polygraph. Or we are no different than voice-stress pimps -selling stuff that ought not to be sold.

Actually, I think you know this already because the MQTZCT technique is designed around carefully devised ideas intended to achieve a high level of test accuracy.

This, is uncomfortably (somewhat) correct:

quote:

---

And Ted, I did wake up and smell the decade. It's still the 1930s. Don't believe me? I suggest that you read John Larson's 400+ page work "Lying and Its Detection," originally published in 1932. Aside from Reid's control question and Backster's standardization of scoring, not that much has changed since then.

---

Ouch.

This is a little like damned if you do and damned if you don't.

On one hand we are using basically the same old sensors - kludged onto a modern laptop. And we still manually score everything - perhaps because we are afraid of statistical "alchemy."

Regardless, I cannot agree with this:

quote:

---

The reality is this: Even the most ardent attempts at 21st-century statistical alchemy will never shore up polygraph's feeble foundation enough to pass scientific muster in the forensic arena.

---

It is the kind of hyperbole that we expect from opponents of the polygraph: Injection of the word "alchemy" to undermine our confidence in the evidence we do have to support polygraph validity, use of the term "shore-up" to create an image of something badly broken, "feeble foundation" which rolls off the tongue as an easily/mindlessly repeated mimetic insult, and the term "scientific muster" as if there is some science-czar that will ultimately approve or dissaprove (there is not - we have to engage the conversation in accurate descriptive detail.

So, if this is not just a fun stick-in-the-eye exercise - if this is an actual conversation about the scientific merits or deficiencies...

What exactly is wrong with the statistical analysis that have been used to describe the scientific validity of the polygraph?

and

What exactly if feeble or deficient about the foundation (construct) of polygraph testing? This one is clearer to me: continued reliance on limited hypothetical formulation that over-emphasize emotion and fear, dependence on a manipulative test administration, TDA models that make little to no use of normative data, fear of statistical algorithms. However, all of these have been and continue to be addressed. More parsimonious hypothesis have been advanced, normative data do exist for validated polygraph techniques, and there is a growing professional population that does understand and make use of statistical algorithms alongside manual TDA models.

The most important thing is to NOT allow our intelligence to be limited to simple hyperbole and sound-bites. That is exactly what our detractors want.

.02

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2012 12:33 PM            

quote:

---

I do not agree that accuracy differs for reasons other than random error.

The recent meta-analysis shows this and states this. Differences in the accuracy rates of all validated polygraph techniques are not statistically significant. This means that we are NOT sure (evidence does not support) that accuracy differs for any reason other than random sampling error (uncontrolled variance).

---

posted 01-09-2012 02:57 PM            

I want to make sure I understand the question correctly. I think you are asking if there is a difference (significant diff) in the variance of the accuracy for the three samples for the RCMP/Utah studies. Is this correct?

I cannot think of any other formulation of the qeustion.

So, we did not calculate the variance or standard error of accurate rates from individual studies. Because criterion accuracy estimates are a simple proportion of decisions that are coded nominally for each sample (i.e., correct, incorrect, inconclusive), it is very difficult to actually calculate the variance or standard error for these statistics. The common approach today is to bootstrap them. However, we did not bootstrap the variance of accuracy for individual studies. This is we were not interested in accuracy estimations from individual studies - that's what meta-analysis is all about (results from single studies are not worth all that much).

We did calculate the standard deviations for the sampling distributions of the individual studies (not the results). Using the procedure described by Cohen, we used the StDevs to conduct two-way ANOVAs to test whether the study samples were representative of each other. (e.g., were the samples drawn from the same population, were the exams administered in a similar maner, were the exams scored in a similar manner). If representative of each other, then there were no signficant diffs in the 2-way analysis. If the 2-way was signficant - then we know mainly that their is something confounding the representativeness of the samples. It is more difficult to determine the origin or cause (e.g, samples from different populations, exams administered differently, or exams scored differently) - we know mainly that something was different.

For the Utah technique, we evaluated the PLC, DLC and RCMP variants in three separate 2-way ANOVAs (2x2, 2x2, and 2x3), and found no difference. We then evaluated the PLC, DLC and RCMP variance in another 2-way (2x3) ANOV A and again found no difference.

The meaning of this is that the three Utah variants appear to produce the sample distributions of sample scores. From that there is every reason to expect that they would also produce criterion accuracy rates that are not significantly different.

Again, we did not actually calculate or bootstrap the standard errors or variance of the results of individual studies, but considering how close the results are, and the sample sizes involved, it would not be significant. It would probably take a very large sample to have a enough statistical power to find those differences significant.

We did calculate the StErr form the accuracy of each technique, and found that they did not differ signficantly (see Figure 17).

Therefore, we conclude that the evidence does not support the notion that differences in accuracy are due to anything other than random sampling error.

r

------------------
"Gentlemen, you can't fight in here. This is the war room."
--(Stanley Kubrick/Peter Sellers - Dr. Strangelove, 1964)

posted 01-09-2012 06:27 PM         

Ray, you can polish the polygraph turd -- using your super-duper 21st-century Willie Wonka computerized number-crunching statistical bootstrapping alchemy machine (complete with Bonferroni turbocharger and kruskal-wallis afterburner) until it shines like gold, but it's still a turd.

Barry, all of your questions/concerns miss my point about the practice test being a ripe business opportunity for the polygraph industry. The practice test simply addresses a need in the marketplace. Whatever value the practice test has is largely in the eye of the beholder, much like a fidelity test. A multiple-issue pre-employment screen is more or less a 50/50 crap shoot no matter how you cut it. There is no moral imperative here. It's strictly business. Do APA-accredited school directors/owners care if their students find meaningful polygraph work or even complete the course? It's all about the tuition checks. This is no different.

posted 01-09-2012 07:03 PM            

Polygraph is a turd, and I have new way to make money with it.

Is this right?

posted 01-09-2012 07:13 PM         

Check your dash board. I think your "gut check" light is on. I could be be wrong. Maybe it is the "service ethics soon" light.

Ted

posted 01-09-2012 09:42 PM         

It's basically a matter of public perception, which, most of the time, translates into a kind of de facto reality.

In the scientific sense, polygraph is definitely seen as a turd. Everyone but those who have something to gain with the polygraph says so. (National/state/provincial/municipal governments, academicians, judges, lawyers, labor unions, scientists, doctors, psychologists, therapists, marriage counselors, even Joe Sixpack -- save the Jerry Springer crew -- are in general agreement that the polygraph is scientifically null and void.)

That's a lot of inertia to overcome.

BTW, I've conducted three focus groups on polygraph perceptions since 2005. (We used a lot of focus groups in marketing back in the day when I was a corporate "bigwhig.") The results of my exercises are discouraging, but maybe the Willie Wonka alchemy machine can turn things around.

Hey, anyone wanna take bets on that? Anyone? No? OK, then. Moving right along...

In the artistic/utilitarian sense, I know polygraph to be effective. Indeed, it "works." Especially with 8th-grade dropouts.

The jury is out on the money making part of it. But the market is there, to be sure, and being exploited, so it appears, as we speak.

Comrade Ted: Thanks for the service reminder. I'll look into it first thing in the morning. Meanwhile, get the chief another
danish -- on the double!

Dan